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result, states and school districts implemented an infrastructure of supports. To date, there is 
little research that considers how support providers use accountability tools to diagnose 
problems and design targeted improvement strategies. Without better knowledge of how schools 
and providers commit to particular improvement strategies, it is difficult to determine whether 
we need better school report cards or strategies, or both to improve school quality. This study 
aims to address this gap by examining how four Children First Networks in New York City used 
accountability metrics to develop targeted improvement strategies, which led to distinctly 
different improvement strategies. The article closes with implications for policy. 
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Resumen: En el Child Left Behind se baso en la logica que midio el rendimiento de la escuela y 
haciendo los resultados publicados a traves de herramientas como las tarjetas de presentation de 
la escuela, animaria a los educadores a crear estrategias para la escuela de la calidad de la escuela. 
Ademas, las estaciones necesarias mas de los incentivos para ser capaces de disenar las mejoras 
de las estrategias que se espera a todos los estudiantes que se aprueban en el estandar como una 
clase. Los resultados, los estados y la escuela destructiva se han implementado en una 
infraestructura de apoyo. Para la fecha, hay poca investigation que considera como los 
proveedores de servicios utilizan las herramientas de responsabilidad para los diagnosticos de 
problemas y el diseno de las mejoras de las necesidades. Si no se sabe mejor como las escuelas y 
los proveedores de un compromiso para las mejoras de rendimiento especiales, es dificil 
determinar si necesitamos una mejor escolaridad de las tarjetas o las estrategias, o bien a la 
escuela de calidad. Este estudio se centra en la direction de esta brecha por el examen de cuatro 
Children First Networks de Nueva York utilizado para medir las respuestas de rendimiento de 
las ciudades, que le permiten diferenciar diferencias de mejora. El articulo de los alrededores con 
las consecuencias para la politica. 

Keywords: Accountability; mejoras; escuelas; Nueva York 

Relacionamentos entre presta§ao de contas e praticas para melhorar as escolas da 
Children First Networks em Nova York 

Resumo: As politicas federais de prestayao de contas da escola, como No Child Left Behind, 
foram baseadas em uma logica que mede o desempenho escolar e tornando os resultados 
publicos atraves de ferramentas como boletim escolar incentivar os educadores a criar 
estrategias para melhorar a qualidade da escola. No entanto, a maioria das escolas precisava de 
mais do que incentivos para projetar estrategias de melhoria que levariam a que todos os alunos 
se tornassem proficientes em assuntos padrao como a matematica. Como resultado, estados e 
distritos escolares implementaram uma infra-estrutura de suporte. Ate a data, ha poucas 
pesquisas que consideram como oferecer suporte aos provedores que usam ferramentas de 
responsabilidade para diagnosticar problemas e projetar estrategias de melhoria direcionadas. 
Sem um melhor conhecimento de como as escolas e os provedores comprometem-se com 
estrategias de melhoria especificas, e dificil determinar se precisamos de melhores relatorios ou 
estrategias escolares, ou ambos para melhorar a qualidade da escola. Este estudo enfoca a 
dire^ao dessa lacuna pelo exame de quatro Children First Networks de Nova York usadas para 
medir as respostas de desempenho das cidades, que permitem distinguir diferenyas de melhoria. 
O artigo encerra com implicates para a politica. 

Palavras-chave: Accountability; melhoria; escolas; Nova York 


Introduction 

It is generally accepted that school accountability is a necessary component of school 
improvement. We rely on accountability to tell us how schools are doing and to show us which 
schools could, or should be doing better. Broadly, there is a belief that accountability could also 
show schools how to improve based on the premise that teachers and principals - often with help 
from a district central office or outside support providers would use data, or feedback from 
accountability tools to develop and enact plans for improving teaching and learning. 

Research has shown that schools do respond to accountability - some in favorable and many 
in unfavorable ways (i.e, Anagnostopoulos & Rutledge, 2007; Booher-Jennings, 2005; Diamond, 
2012; Diamond & Spillane, 2004; Firestone & Gonzalez, 2013; Llanushek & Raymond, 2004; 
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Mintrop & Trujillo, 2007; Newmann et al., 1997). Specifically, these studies investigate how schools 
respond to conventional test-based accountability models and conclude that improvement efforts 
are unpredictable, uneven, and largely attributed to a school’s cultural and professional resources 
(Diamond, 2012; Firestone & Gonzalez, 2013; Newmann et al., 1997). 

The evidence leaves little guidance to states and districts attempting to create new 
accountability systems as called for by The Every Student Succeeds Act (ESSA) that was signed into 
law in 2015. ESSA requires states to redefine their accountability systems to include alternative 
indicators of progress and school quality, as well as, create systems of support for those schools that 
are not meeting a State’s standard for performance. ESSA presents an opportunity for states to 
create more balanced systems of accountability and support that could lead to more favorable 
responses to accountability and improve the quality of instmction for all students. Yet, to date, there 
have been few studies that provide clear insights into how metrics become improvement practices. 

Prior research tended to focus on the effects of accountability policies using changes in 
students’ test scores to determine whether a policy worked as intended, or not (i.e., Dee & Jacob, 
2010; Hanushek & Raymond, 2004; Kane & Stagier, 2002). The evidence is mixed as to whether 
accountability actually works as a way to improve all schools. One explanation for this is that the 
policies expect all schools to meet the same standards for school performance without really 
translating what capabilities must be developed or what kinds of changes to organizational 
conditions would be necessary in order to satisfy what policies demand (Finnigan and Daly, 2012; 
Massell & Goertz, 2005; O’Day, 2002). This leaves the work of diagnosing problems and designing 
solutions to those working in and with schools (Spillane & Coldren, 2011). Much of the research 
was based on economic methods and concluded that there was variability in outcomes, but the 
studies did not investigate practice, or the interactions among educators as they constructed 
problems and created solutions (Coburn, 2006), which meant it was difficult to discern how 
practices varied within schools and what mechanisms promoted or inhibited changes in educators’ 
practices. 

Previously, those scholars who did investigate how schools diagnosed and designed solutions 
were confined to examining test-based models that judge school performance on students’ test 
scores in select subjects. These metrics were disconnected from schools’ internal conditions like 
routines, norms, and formal and informal social relationships that are the mechanisms through 
which educators collectively diagnose problems and commit to solutions (Finnigan & Daly, 2012; 
Mintrop & Trujillo, 2007; O’Day, 2002; Spillane et al., 2011). As a result, it was difficult to discern 
whether the unevenness of schools’ responses that scholars were finding was due to the varying 
quality of schools’ social relations, stmctures or professional cultures (Bryk & Flermanson, 1993; 
Bryk & Schneider, 2002; Bryk et al., 2010; Elmore, 2004; Finnigan & Daly, 2012) or the 
shortcomings of test-based accountability models (see Kane & Staiger, 2002; Koretz, 2008). 

This study aims to offer evidence from an exploratory study of New York City’s 
accountability system that when it was implemented was a radical departure from conventional test- 
based models used in most school districts across the United States. In 2007, New York City’s 
Department of Education (NYC DOE) implemented two novel instruments to hold schools 
accountable, the Progress Reports, which relied on some conventional quantitative metrics and 
some pioneering metrics to measure progress, and the Quality Reviews, which measured schools’ 
organizational conditions related to instructional improvement. In addition, the NYC DOE 
organized new school support teams, the Children First Networks (CFNs), which replaced regional 
district offices. These teams did not have any supervisory authority over empowered principals; 
rather, they were meant to be genuine partners with their schools. Through joint efforts, the CFNs 
and schools would be able to develop each school’s continuous improvement practices and increase 
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school quality. This set of reforms was specifically designed to promote and enable school leaders to 
diagnose and design targeted solutions that would meet each school’s needs (Childress et al., 2010). 
Through an investigation of how the CFNs used the two accountability tools in their relationships 
with schools, this study demonstrates how accountability metrics become constituted in practice by 
documenting the ways in which school support teams selected one tool over the other to set 
priorities and negotiate customized plans with their partner schools. 

As States and districts attempt to respond to the Every Student Succeeds Act (ESSA) and 
design new forms of accountability for schools, these findings provide timely insights into the 
conditions, relations, and practices involved when external accountability measures are used with 
diverse schools and become part of improvement efforts. The sections that follow describe the 
context of this study and present descriptions of the conceptual framework, methods used for this 
study. The final section presents implications for policy. 

Context 

In 2002, New York City’s Mayor Michael Bloomberg and Schools Chancellor Joel Klein set 
in motion a series of reforms to overhaul the city’s school system. These leaders sought innovation 
and improved performance — neither of which had been incentivized under the former system. They 
pursued these goals using a variety of instmments including a novel accountability scheme and 
school support stmcture. 

In 2007, the New York City Department of Education (NYC DOE) designed and 
implemented two instruments to hold schools accountable: Progress Reports and Quality Reviews. 
Progress Reports relied on quantitative measures like student achievement on tests and school 
climate surveys to give schools an annual grade, A-F. Quality Reviews involved school visits in 
which expert educators walked through classrooms and sought to assess school quality using a 
mbric that was based on three domains: instructional core, school culture and systems for 
improvement. While visiting the school, the reviewers also met with parents, teachers, and school 
leaders to ask questions related to the mbric. School leaders wrote a self-evaluation of their schools. 
Using the leader’s self-evaluation and the evidence they gathered during their site visits, reviewers 
determined the extent to which a school was equipped to improve instruction. Schools received one 
of four possible Quality Review designations: Well Developed, Proficient, Developing, or Under¬ 
developed. 

Schools received Quality Reviews on a staggered scheduled. Unlike Progress Reports that 
were published every year, all schools underwent a Quality Review, at least, once every four years. 
Some schools received reviews more often based largely on criteria the DOE developed over time. 
For example, one rule specified that if a school got a C on the Progress Report for three consecutive 
years, then that school would be targeted to receive a Quality Review in the subsequent year. If a 
school was labeled as Developing or Under-Developed on the Quality Review, then the subsequent 
year, the school would receive a Developing Quality Review, which was a review without stakes. The 
school had one year reprieve, but in the following year, the school would receive a standard Quality 
Review that included stakes. 

In combination with the accountability tools, the NYC DOE changed the support stmcture 
for schools — moving away from a regional structure under which superintendents were responsible, 
but not accountable, for providing support to schools and checking compliance with mandates. 
Bloomberg’s DOE replaced the regional superintendents with a network stmcture that resembled 
market-based service organizations in which principals selected which support team, referred to as a 
Children First Network (CFN) from which they would receive comprehensive supports. 
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From 2010-2014, all schools in NYC, nearly 1,700 schools, including public and charter 
schools received instructional support from a CFN team. Network teams were comprised of 
approximately 15 people who were experienced educators and former school leaders. Unlike 
superintendents before them, Networks lacked authority to require schools to engage in particular 
activities. Rather, Networks were viewed as service providers to schools - charged with helping 
schools by targeting support to each school’s unique needs. 

Annually, the NYC DOE evaluated the Networks and ranked them according to the 
performance of the schools in the Network. A Network could be disbanded, if its scores were too 
low, or the number of schools in the network dropped below 18. By design, the Networks were held 
accountable through a combination of bureaucratic and market methods, but there were few 
instructions about how they were to fulfill their charge and little guidance to help them learn how to 
provide targeted support. This study investigated how Networks sought to set priorities and engage 
in improvement practices with their partner schools while they were under pressure from both the 
NYC DOE and principals to add value to schools. 

Conceptual Framework 

Accountability policies assume that incentives and pressure will encourage schools to engage 
in continuous improvement practices by using student-achievement data to detect and correct 
errors. Another way to consider this is that accountability policies expect schools, or school leaders, 
to be able to diagnose the nature or cause of problems within their local conditions and to make a 
prognosis that includes setting goals and developing strategies for remedying problems that have 
been identified (Benford & Snow, 2000; Spillane & Coldren, 2011). 

In complex environments like schools, the root causes of problems related to student 
learning are likely ambiguous or they may be linked to systemic issues in society like poverty, weak 
professional structures or misaligned instructional systems that reach inside schools and affect how 
people make sense of their circumstances (Cohen & Moffitt, 2009). As a result, the work of 
diagnosing problems related to student learning or school performance may not be straightforward 
(Peurach, 2011; Spillane & Coldren, 2011). Further, “problems do not exist as social fact awaiting 
discovery” (Coburn, 2006, p. 343). Rather, problems are constmcted by school leaders and support 
providers charged with helping schools improve. Constmcting problems, or as policy scholars have 
referred to it as problem framing happens as school leaders and support providers interpret and 
attend to certain aspects of their social world, while they ignore others (Coburn, 2006; Spillane & 
Coldren, 2011; Spillane & Miele, 2007; Weiss, 1989). The process of framing a problem assigns 
responsibility and creates a rationale for certain strategies over others (Coburn, 2006). 

Diagnostic framing influences prognostic framing, or the solutions that are articulated as 
plans or strategies for addressing what has been identified as problems (Benford & Snow, 2000). 
Although, prognosis does not always follow from problems; rather, sometimes schools take 
preventative actions, or follow mandates that require them to implement particular programs or 
interventions that were selected before a problem had been defined. Taken together, these two 
processes, diagnostic and prognostic, form the basis of designs for improving teaching and learning 
(Spillane & Coldren, 2011). 

These concepts of diagnosis and design for school improvement (Spillane & Coldren, 2007) 
form the foundation for this study. How these processes unfold for schools and support providers 
responding to accountability pressures builds on what we know about external support for school 
improvement and how support providers use accountability tools as boundary objects to orient their 
partnerships with schools. 
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External Support 

Recent research on external support for school improvement has focused on the scaffolds 
and resources that support providers, often referred to as brokers, provide to support learning 
through social interactions and collective sensemaking (Honig & Ikemoto, 2008). Seen as “learning 
supports” these engagements are meant to build schools’ organizational capacities to change 
teaching, which itself is a problem of learning for central office and support providers (Cohen, 

1982). Drawing on sociocultural learning theory, Honig & Ikemoto (2008) coined the term “adaptive 
assistance” to characterize how the Institute for Learning (IFL) at the University of Pittsburgh 
worked with districts to build capacity to support changing instruction across schools. Adaptive 
assistance relationships use a variety of social activity stmctures to promote targeted and authentic 
learning that fits local context. These social stmctures include modeling, facilitating dialogue, 
developing and elaborating tools, and emphasizing joint-work focused on specific problems of 
practice (Honig & Ikemoto, 2008). The authors refer to these activities as adaptive because, 
“participants in those assistance relationships do not simply replicate behaviors of the past, but 
continually assess their situations (especially to the extent to which those situations are routine, or 
nonroutine); take action; and revisit the fit between their goals, actions and outcomes” (Honig & 
Ikemoto, 2008, p. 354). This orientation viewed support as dynamic and emphasized tailoring 
support to fit the needs of districts, or schools, that were receiving support, which stands in contrast 
to conventional support that may be predetermined or focused on program-specific learning where 
the goal is to impose a set of ideas on a school regardless of site-specific context. 

When support is not targeted to schools’ local realities, the mismatch between the ‘help’ 
offered by district offices and the day-to-day practices happening inside schools has created 
situations where schools are faced with the additional burden of realigning support that is less than 
helpful — or worse, support that compounds or complicates matters. Rather than helping schools 
make progress in developing capabilities, support that is overly general, or one size fits all, can 
actually detract from the main objectives of improving teaching and learning (Finnigan, O’Day, & 
Bitter, 2009; Honig & Hatch, 2004). 

To counter the one size fits all approach, some districts have begun offering what Honig et 
al. (2010) referred to as project-management school support. With project-management support, 
cross-department teams work with schools to identify specific problems that are directly relevant to 
particular school contexts. Rather than having centralized support that comes from siloed 
departments dealing with budgets, human resources, safety, and instmction separately, these 
dynamic teams bring staff from multiple departments together strategically. Working with school- 
based leaders, the teams collaborate to diagnose problems and create solutions. 

In their analyses of several districts, Honig et al (2010) found that project-management 
support structures are better equipped to deal with the complexity involved in any one problem that 
a school may encounter because multiple sets of expertise are brought to bear on solving a problem 
that the group identified as such. For example, a school may want to offer additional tutoring 
services for certain students after-school because, as the team diagnosed jointly, test score data 
revealed a there was a problem for those specific students, but finding the funding and knowing 
how to navigate budgets required particular expertise that often resides within specific departments 
at central. Siloed support from central office can create incoherence for schools, or incomplete 
solutions, which may or may not be aligned with a school’s given goals for improvement (Ikemoto 
& Honig, 2010; Spillane, 1998). 

The advantages of using a project-management approach is that it created novel 
relationships between central office staff and school-based leaders that were focused on 
collaborative problem-solving - instead of compliance-based relations that, at times, complicate 
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matters for school leaders who, when dealing with siloed support, have to reconcile various 
messages they may receive from various offices. When leaders have to reconcile, or synthesize 
messages, they are left with the burden of crafting coherence for themselves and their school 
community (Honig et al., 2010). 

Whether support operates like an adaptive assistance relationships, or project management 
team, the providers must navigate the tensions around the competing demands to direct and co¬ 
construct work with partner schools. After all, the schools are working with providers because they 
need support and may need direct guidance. There is still more to be learned about the nature of 
social relations between providers and schools and the conditions, like accountability structures, that 
hinder or enable more productive partnerships (Honig & Ikemoto, 2008). 

Orienting Joint Efforts with Boundary Objects 

Scholars have noted that one way support providers have tried to manage the ambiguity 
involved in adaptive relationships is to rely on “boundary objects” or “concrete objects that embody 
a set of ideas or processes” that serve as a “point of focus” for providers working across different 
contexts (Stein & Coburn, 2008, p. 589). One example of a boundary object, according to these 
authors, is pacing guides. Coaches introduce the same pacing guides to teachers in different schools, 
and in turn, those teachers may meet at district-wide meetings to discover and learn from how other 
schools use the guides. Pacing guides, or boundary objects more broadly, “have the potential to 
coordinate perspectives and spur similar forms of learning across multiple communities” (Star & 
Griesmer, 1989, as cited in Stein & Coburn, 2008, p. 598). 

According to Wenger (1998), learning, or changing how people work, individually or 
together, involves opportunities for interacting, participating, engaging with objects, or reifications, 
toward some shared purpose. In this way, practice is learning, and learning is practice, but it involves 
looking at the coordinated, patterned, and meaningful interactions of people at work, that is 
shapeded, in part by their contexts, which includes artifacts (Coburn, Bae, & Turner, 2008; Cook & 
Brown, 1999; Spillane, 2012; Stein & Coburn, 2008). 

Some have referred to the conditions and resources that shape the nature of group 
interactions as an architecture, or infrastructure that influences how, when and why people engage in 
social constmction of plans, develop new practices, or create new shared commitments to particular 
strategies for improving classroom instruction (Stein & Coburn, 2008). Looking at two districts’ 
architectures for learning, Stein & Coburn (2008) found that even when both districts designed 
artifacts and provided professional development to help teachers learn about more effective teaching 
practices, the “opportunities to learn are shaped by how meanings are negotiated surrounding 
particular boundary objects” (p. 617). In one district, the types of boundary objects that the district 
used to help teachers learn about a new curriculum ended up constraining teachers’ engagement and 
limited their ability to negotiate meaning with each other. For these teachers, it appeared that there 
was ‘one right way’ to implement the curriculum, which left little room for social interactions. The 
architecture in this district did little to promote changes in practice. Alternatively, in the other 
district, the curriculum and designs for learning offered “ample openings for individuals to negotiate 
meaning” (p. 617). The authors concluded that “the nature of boundary objects used by district 
leaders matters” because it influenced the kinds of engagement that happened in social contexts 
(Stein & Coburn, 2008, p. 617). 

Contexts including who interacts with what objects and the designs of those objects and 
under what circumstances the social interactions occurred appeared to matter for degree of learning 
that took place and the extent to which changes in practice would be improvements (Honig & 
Ikemoto, 2008; Stein & Coburn, 2008). 
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Most research, to date, has considered boundary objects related specifically to instruction or 
curriculum reforms - not necessarily score cards, or mbrics used to evaluate schools (i.e., Cobb et 
al., 2003; Coburn, 2001; Stein & Coburn, 2008). Unlike curriculum pacing guides, scores cards and 
mbrics may carry stakes, which changes the nature of the ways in which people may attend to them. 
The popularity of reform like Portfolio districts, which close schools based on accountability metrics 
(see Hill, 2010), make score cards and rubrics particularly important objects. We do not have a sense 
of what support providers do or the role they could play in helping schools develop continuous 
improvement practices when they are held directly accountable for how their partner schools 
perform as was the case in New York City between 2007-2014 and other portfolio districts. 

This study attempts to extend studies that have considered curricular reforms that 
investigated how districts and schools tried to strike a balance between reification and participation. 
Instead of curricular materials or standards focused on one subject, this study considers novel 
accountability instmments as the boundary objects. The objects in this case, the Progress Reports 
and Quality Reviews articulated particular meanings of what would constitute improved whole school 
performance and not just uptake of a particular curriculum. Then, I considered the social 
interactions as school leaders and support providers interacted with these instmments to negotiate 
meanings, diagnose problems, and craft prognoses or tailored strategies for improvement. 

Data and Methods 

This research draws from a larger study that investigated the design and use of New York 
City’s Progress Reports and Quality Reviews. The initial purpose was to understand how the 
architects of the reforms sought to influence school improvement by changing system-level 
structures. The aim of these reforms was to increase the quality of schools and the number of high 
quality schools in the district by incentivizing school leaders to become strategic and innovative 
managers of continuous school improvement. According to Chancellor Joel Klein, “We are 
transforming a school system based on compliance and top-down decision making to one that 
empowers principals to make decisions about what is best for students and their school 
communities” (NYC DOE, 2007, p. 1). Knowing that most school leaders would not be able to do 
this on their own, an innovative school support stmcture, the Children First Networks (CFNs) was 
established to enable innovation and changes in school improvement practices at the school level. 
The focus of this study was on the strategies that emerged from the partnerships between CFN s and 
schools and how accountability tools influenced commitments to particular strategies. 

Data Collection 

Data for this research were collected between June 2013 and July 2014, which was the final 
year of Mayor Bloomberg’s administration and come from interviews with current and former 
district administrators («=20) and Cluster team members, who were charged with supporting all of 
the eleven CFNs in the Cluster ( n-7 ). I interviewed all CFN team leaders from four focal CFNs 
(«=4) and coaches from these teams («=5). In addition, I interviewed principals of schools that 
belonged to two of the focal CFNs (n- 4). These principals were selected based on 
recommendations by network team members because of innovative leadership, or due to marked 
changes in a school’s Progress Reports or Quality Reviews. 

Interviews typically lasted 60 minutes and were conducted using a semi-structured protocol 
that was developed by the author. The goals of the interviews were to learn about the aims of 
accountability system in New York City, features of the Progress Reports and Quality Review, how 
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the tools were used by central office and CFNs, school support structures, and resources for 
implementing the tools. All interviews were recorded and transcribed. 

Data also come from field notes from observations of Network Leader Institutes, which 
were all-day meetings facilitated by central office («=4), Cluster meetings {n- 13), CFN team 
meetings and meetings with principals («=6). In addition, there are notes from observations of 
Network Performance Management meetings for each of the focal networks during which network 
leaders presented to the Cluster team an overview of the supports the CFN team had provided to its 
schools throughout the year. 

Table 1 provides an overview of the four focal CFNs that were the subjects of this study: 
CFN 701, 702, 703, and 704. 1 I focused on these four Networks because they were recommended 
by the Cluster leader and considered within the Cluster and through citywide comparisons to be 
successful support teams. Successful teams were ranked higher on the Network Performance 
Management system; they were more likely to spend more of their time providing support to 
schools as opposed to other CFNs with steeper learning curves that spent time more learning how 
to be a CFN. As the Cluster leader mentioned, “Not all networks are created equal.” Finally, the 
focal CFNs were selected because they actively pursued relationships with schools and tried to 
customize supports, which provided access to how the CFNs and schools negotiated commitments 
to particular improvement strategies. 

Table 1 


Description of Focal Children First Networks (CFNs) 


Pseudonym 

Number of schools 

Location of schools 

Types of schools 

701 

34 

Manhattan (3) 
Brooklyn (15) 
Bronx (14) 
Queens (1) 

Staten Island (1) 

Elementary and PK-5 

702 

26 

Manhattan (4) 
Brooklyn (6) 
Bronx (11) 
Queens (5) 

High schools 

703 

30 

Manhattan (8) 
Brooklyn (12) 
Bronx (6) 
Queens (3) 

Staten Island (1) 

High schools 

704 

33 

Manhattan (14) 
Brooklyn (7) 
Bronx (6) 
Queens (5) 

Staten Island (1) 

Middle and elementary 
schools 


As seen in Table 1, the CFNs were not geographically based; rather, each of the focal CFNs 
served schools in several boroughs. CFN 701 had office space in the Bronx, CFN 702’s office was 


1 These are pseudonyms for the CFNs. 
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located in Manhattan and the other two teams, as well as, the Cluster had offices located in 
Brooklyn. The CFNs had distinguishing features that set them apart from the other networks. 

The Network leader for CFN 701, according to the Cluster leader, was a “fierce 
administrator” who was known for managing her team and resources well. CFN 702 was known for 
its novel approach to delivering professional development for principals. The CFN 702 network 
team was staffed by former directors who worked for central office managing the development of 
the Quality Review Process for the city. One member from CFN 702 had championed the quality 
assessment process the DOE used to make sure reviews were accurate and reliable; she was 
responsible for training those who served as reviewers. Tier experiences leading the development of 
the Quality Review informed how the CFN thought about its relationship with schools. 

CFN 704 was led by a strong instructional leader, who previously served as a principal and 
coach; she was known for being demanding and holding high expectations of her team members. 
She believed that the team’s job was to make sure that all schools - regardless of neighborhood, 
were offering the kinds of instruction parents could expect from elite private schools on the Upper 
East Side of Manhattan. Serving mostly elementary schools, CFN 704 was focused on building 
strong relationships with all thirty-three of the principals in the network that came from all five 
boroughs. Lastly, CFN 703 was a network that was known for “moving principals” and getting 
quick results. In 2013, CFN 703 improved its ranking in the Cluster going from being ranked fourth 
among eleven to first in the Cluster because the schools in CFN 703 had markedly improved their 
Progress Report grades in one year. 

Data Analyses 

Drawing on the concepts outlined by the conceptual framework, the analyses for this study 
were focused on what Wenger (1998) refers to as “design for practice,” which involves “decisions 
about how to distribute a design between participation and reification - what to reify, when, and 
with respect to what forms of participation” (pg. 232) and the concepts of “diagnosis and design for 
school improvement” proposed by Spillane & Coldren (2011), I analyzed the data to understand the 
meanings that were negotiated through the Networks interactions with their partner schools and the 
designs for practice, including diagnoses and prognoses, that they selected to pursue as a result of 
these negotiations. 

Data analyses occurred concurrently with data collection in an iterative manner (Miles & 
Huberman, 1994). I used Dedoose Qualitative Software to organize and code interview and 
observation data and excel to trace messages outlined in artifacts. Throughout the course of the 
study, I wrote descriptive and analytical memos to focus my inquiry towards answering my research 
questions (Miles & Huberman, 1994) that were based on multiple sources of data, which was made 
possible by Dedoose’s feature that allowed me to link several sources to memos. 

I developed a priori codes to reflect ideas about design and use of the Progress Reports, 
Quality Reviews, and the development of plans for supporting school improvement. The initial 
codes were used to identify how the CFNs used the different tools and how they used them with 
their partner schools. In subsequent phases of data analyses, I revised the codes to reflect my 
emerging understanding of the different uses of the tools and to qualify categories of customized 
support as defined by participants, as well as, network-wide types of support. I compared the 
differences and similarities of the various support stmctures and uses of the tools. 

Whenever possible, I relied on multiple sources of data to guard against threats to validity 
(Maxwell, 2005), which included drawing on external reports written about the CFNs. During in- 
person, regular meetings that I scheduled with the Cluster leader and two of the network leaders to 
talk about the study, what I was learning and to stay informed about Cluster and network schedules. 
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I used informal member checks to confirm my emerging sense of networks and the varied ways they 
sought to support schools (Lincoln & Guba, 1985; Maxwell, 2005). 

Findings 

The Children First Intensive was a massive reorganization of New York City’s Department 
of Education that began in 2003 with Mayor Bloomberg’s takeover of the city’s school system. By 
2010, there were 55 Children First networks referred to as CFNs. Principals were encouraged to 
select a network based their school’s priorities and goals. For example, schools that focused on 
project-based learning could join a network that provided specific pedagogical supports relevant to 
project-based learning, or a school may elect to join a network that targeted the needs of middle 
school students, or specialized in STEM, or affiliated with a charter network like Achievement First. 
This market stmcture created incentivizes for networks to be attractive options for schools - as the 
schools could go elsewhere, if their local options were not a good fit. 

In addition to a market, the networks faced bureaucratic pressures from central office 
because they were held accountable for how well the schools in the network faired on the Progress 
Reports and Quality Reviews. This dual-accountability structure put pressure on the Networks to 
develop reciprocal partnerships with schools from which both the school and Network would 
benefit. If a Network failed to retain schools, or their schools’ scores dropped too far, a Network 
would be closed and team members would be without a job. 

To negotiate a partnership with its schools, a network had to figure out how to set priorities, 
identify courses of action, and establish plans for improvement that the school and network would 
pursue together to improve school performance. In some instances, a network team would take the 
lead in these negotiations and in others, the networks would contribute to, but not manage 
negotiations; they would defer to the school. A network’s success depended on how well it could 
customize its partnerships with schools and encourage schools to engage in improvement practices 
that would get results. 

To work within these novel social arrangements, the Networks that I observed attempted to 
assume the role of information-manager. They tried to manage information about the district’s 
priorities (including the accountability tools, new ELA and math curriculum, Common Core 
Learning Standards, and new data management systems). This effort was described by the Network 
leader from CFN 701, 

You gotta keep up with stuff because nobody tells you. Sometimes, you have to learn 
and find out on your own.. .you start to read and figure things out; then, you make a 
DOE, a central, or TWEED training. Cluster will give a training.. .but you always try 
to keep on top of things because sometimes things happen and you don’t get the 
information right away because they [Cluster] is trying to digest it, but you can’t sit 
and wait two or three weeks because questions are coming your way about this. 

People are hearing and you’ve gotta try to figure out, ‘how am I going to answer 
this?’ or ‘What do I tell people?’ 

In this excerpt, the Network leader identified what it meant to be responsible for managing 
information about district initiatives and needing to figure out how to help schools get the 
information that they needed. Similarly, the Networks needed to gather information about their 
schools so that they could develop targeted strategies of improvement, as well as, cultivate 
relationships with schools to maintain their partnerships. They had to walk the line between expert 
and learner. 
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Needing to be adaptive and flexible, while also presenting schools with the information and 
resources that they needed, the networks worked to develop capabilities to learn quickly; they 
learned to prioritize and narrow their search for information. To accomplish this, the Networks 
relied on the accountability tools, the Progress Reports and Quality Reviews as a way to determine 
relevancy when they sought information, and as a way to legitimize the information that they 
delivered, as well as, inform their search for information about their schools. 

Consultations: Diagnosing and Crafting Solutions 

From interviews and observations, I learned that the common practice the Networks used to 
negotiate the terms of their partnerships with schools was referred to as a consultation. Consultations 
were meetings between Network team members and school leaders, and whomever the principals 
wanted to include (i.e., Assistant principals, school leadership team members). Typically, 
consultations took place in May or during the summer, usually in August as they were treated as the 
time and place to develop plans for the following school year. There were no mles pertaining to 
these meetings, or protocols that dictated how they should happen, but they were used when a 
Network was connecting with a principal for the first time, and Networks held consultations with 
principals with whom the Network leader may have had a long-term relationship that may have 
roots going back to pre-Network days when a school board, not the Mayor, was in charge of the 
city’s schools. The consultations, even though they were not required became standard practice 
among the Networks that I observed. 

Consultations emerged as an important opportunity for a Network to visit a school, meet 
with the leadership and develop a strategy for supporting the school going forward. This standard, 
but informal practice was described by a Network Leader from CFN 704, 

.. .this is an informal way of us just deciding, by meeting with them [principals] in 
the summer.. .we use this to do our planning.. .this helps us develop our network 
plan. I think that most networks do it that way, but it’s not something that’s 
required. Sometimes they [networks] don’t have conversations; they just wait for the 
principals to tell them what they need, but I’ve never felt that was good - 
sometimes, principals don’t know what they need, and I think by having these 
conversations, and by looking at data together and looking at trends and patterns 
and then thinking about it, I think that that’s much more successful in trying to get 
these goals accomplished and in moving student performance. Once again, I don’t 
think I’m the only network that does it that way. I’m sure there are others, but it’s 
not something that’s required. 

In this excerpt, CFN 704’s Network Leader highlights how she chose to meet with her principals to 
determine plans, even though, “it’s not something that’s required.” Rather than “wait for the 
principals to tell them what they need,” she and her team used the data from the Progress Reports 
(trends) and Quality Reviews in conversations with principals to advance their shared goals of 
improving school performance and “moving student performance.” The consultation allowed the 
Network to “do our planning.” 

Consultations were described as taking place in advance of the school year (usually in the 
spring or summer) and were used as a place for the Networks to learn about a school’s goals, or help 
craft the goals and make plans based on the specified goals. These were negotiations among 
Network team members - usually a Network leader, and school leadership - usually a principal. In 
an interview, a leader from CFN 704 explained how her team used consultations. 
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What we try to do is every August, in the summer, we meet with our principals and 
their cabinet and we look at the Progress Report. We’ll see that as far as progress, the 
students are making progress so whatever interventions and whatever they’re doing 
with Level Ones going to Level Twos is definitely strong - and we want to continue 
with that. We’ll look at subgroups to see maybe if there are certain subgroups though 
that are not making adequate gains so we may think about what are we doing for our 
L populations? What are we doing for students with disabilities? And then talk about 
what the goals are: What are the programs? Who’s teaching those classes? And 
looking at all that. And then we’ll look at the performance and then we will target 
that and say, okay great, you’re an A school, you have a great environment, the 
culture is good, you’re moving your Ones to Twos but you’re not getting those 
Threes and Fours so what are we doing here? What’s the rigor? What can we do for 
our Twos to get them to Threes? We’ll talk about what supports we can give them, 
what they’re doing in the school, so we will always break down the entire report and 
not just say, ‘you’re an A school, so you’re fine.’ So we have to look at the realities of 
it and that’s the way we look at that. And then we’ll bring in the Quality Review, as 
well, and look at what those indicators are telling us, and I think I like to look at that 
to see what’s in the classrooms and what’s happening as far as teaching and learning. 

The leader from CFN 704 described how she and her team consider a school’s Progress Reports and 
Quality Review to assess the how students are doing in a school - including how different sub¬ 
groups are doing. They investigate and consider the interventions already in place in a school and 
consider a school’s goals. They ask, “what can we do for our Twos to get them to Threes?” The use 
of “we” suggests that the Network tried to position itself as a partner in improving student 
performance and targeting each school’s area of need. This could also be seen in the leader’s 
description of how the network would, “.. .talk about what supports we can give them and what 
they’re doing in the school.” 

In her description of a hypothetical consultation, the leader from CFN 704 highlights an 
important feature of the accountability scheme, which was that even conventionally successful 
schools, or those that received an “A” on the Progress Report were not “fine” or immune from 
needing to work on improving student performance and instruction. She says, “we will always break 
down the entire report [Progress Report] and not just say, “‘you’re an A school, so you’re fine.’” 
Rather, all schools were to be working towards improving matters for the lowest-third of students 
(which all schools have), as well as, working towards making sure that there was rigorous instmction 
happening in all classrooms. The consultations were used to surface the school’s view and a 
Network’s interpretation of a school’s areas that needed targeting so that a plan for interventions 
could be developed. 

It was during the consultation meetings that the Networks best demonstrated their varied 
uses of the accountability tools — agilely providing information to schools and gathering information 
about their schools from the school’s perspective. The Network Leader from CFN 701 described 
how her team managed these tasks, which highlights the multiplicity of the tools in consultations. 
Once we sit down initially, and usually Sheila [deputy leader] and I will be at the first 
meeting where we kind of give the overview, then we begin to ask questions around 
the specific indicators. And in taking in their responses and where we see where the 
work needs to be, then we’ll sort of tailor, customize, cause everyone’s in a different 
place. 
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Here, when the Network Leader from CFN 701 refers to the “overview,” she is talking about the 
overview of a school’s Progress Report data, and when she says that they asked questions about the 
“indicators,” she is referencing the Quality Review rubric. 

As a reminder, the Quality Review rubric evaluated schools using 10 indicators grouped into 
three domains: School culture, stmctures for improvement and instructional core. The three 
domains related to the instmctional core were worth double points and likely to be targeted during 
discussions between the network teams and schools. One question the team would ask is how the 
school aligns its curricula to the Common Core Learning Standards for all students (Indicator 1.1, 

NYC DOE, 2014). Another may consider how and when teachers meet in teams and whether it is 
structured time that aligns with the school’s instructional goals (Indicator 1.3, NYC DOE, 2014). 
Based on the responses the Network team members hear from their school partners, they will 
“tailor, customize” support plans to fit each school’s needs. 

Customizing Support Plans with Accountability Tools 

One of the touted strengths of the Network support stmcture was that it permitted schools 
and Networks to develop customized plans for improvement based on each school’s needs. In this 
kind of structure, I expected to see among the four Networks that I observed that they would 
develop customized plans and use variety of strategies to support improvement across the seventy 
schools that belonged to these Networks. While there was variation as Networks tried to customize 
plans, what was striking was how common it was to use the accountability tools as frameworks for 
defining the terms of engagement with their schools. The variation depended upon whether a 
Network privileged the Quality Review or the Progress Reports. There were assumptions and 
tradeoffs with each approach, and the most important finding was that customized support looked 
different across Networks - not because the schools were different, but because the Networks were 
using one of the accountability tools to prioritize a course of action that matched a specific 
accountability tool and the school’s circumstances. In turn, what constituted improvement practices 
varied because Networks tended to favor either the Progress Report or the Quality Review, which 
specified different timelines and focus. Focusing on the Progress Reports called for a short-term 
approach to improvement and mainly engendered commitments to certain students; by contrast, 
when Networks used the Quality Review to dictate plans for improvement, the outcomes were more 
likely to be long-term commitments to the development of routines and practices that would alter 
instruction and cultivate a culture of continuous improvement at the school level. In the sections 
that follow, I illustrate these two common approaches to customizing plans for improvement. 

Using the Progress Report in a short-term, numbers game. Progress Reports were 
annual measures, which meant every October a school was exposed to public scrutiny from parents 
and community members, as well as pressure from the Department of Education about its grade, A 
- F, on the Progress Report. Under Mayor Bloomberg, the NYC Department of Education had 
planned to close 10% of the lowest performing schools (approximately 40 schools) annually. 

Because the DOE wanted schools to focus on trying to continually improve instmction, 
performance was not based on absolute test scores; rather, performance was based on several 
factors - including relative comparisons to what were considered peer schools, a group of 
approximately 40 schools that served similar populations of students. This pressure, combined with 
the public nature of annual report cards meant there were strong incentives for schools to pay 
attention to the numbers (and grades) on the Progress Reports. 

There were some Networks that assumed that moving a school’s numbers on the Progress 
Report was the most important goal of improvement practice. This was in part because with a solid 
understanding of the Progress Report’s methodology, schools could make simple changes and see an 
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increase in their scores. For example, because of the methodology, it was possible that a school 
could graduate three more students in one year and beat one of its cohort schools, which would lead 
to a boost in that school’s scores. By comparison, trying to change instruction in classrooms - 
something a principal can, at best, indirectly influence could appear like a long-term investment that 
may or may not lead to higher scores on the Progress Report. Eventually, research would 
demonstrate that schools that focused on the Quality Review would see increases on the Progress 
Reports, but principals and Networks were not always aware of the evidence (see NYC DOE, 2013). 

These conditions created an environment in which it appeared that the greatest service a 
Network could provide to schools was to help principals understand the Progress Report 
methodology as a strategy for improving school performance - the impact of which would be visible 
in higher scores on the Progress Reports. Because the Progress Reports were so visible to 
community members and seemed to be an important tool in a pressure-filled portfolio district that 
Mayor Bloomberg and Chancellor Klein were leading, it could appear as though good grades on the 
Progress Report was comparable to profitable returns for businesses. One Network leader described 
this approach as being, “well, I just want to say, it was so well received.” 

One former network leader from CFN 703, who was working on the Cluster team, explained 
to me that as a network leader, he helped principals identify the things believed to be under the 
principal’s purview. It was believed that, if a principal was proficient in managing data and managing 
information, then he or she could target policies or interventions in such a way that it would allow 
the school to earn an A or B on the Progress Report. Earning a C on the Progress Report for one 
year was not associated with any consequences, but getting a C three years a row meant that a school 
was subjected to sanctions - one of which was a change in school leadership. The former Network 
Leader for CFN 703 explained, “.. .As and Bs - that is the most important... The instructional work 
of the Quality Review, well that is longer term work and there’s teacher turnover. If you get an A or 
B on the Progress Report, it doesn’t matter what you get in the QR. It’s not hard to get a B, if you 
figure out which pieces are within your control.” 

The Network Leader from CFN 701 also described helping principals determine which high 
leverage indicators on the Progress Report they should focus on to gain the greatest advantage. She 
explained, 

Our work is to help [principals] move their school. And we’ve been pretty successful, 
at least, we had an F last year but they closed, the Department of Ed due to whatever 
input they, and I felt like the leader, which was brand new, it was a first-year 
principal, so she had just been putting things in place and whatever, but they decided 
to phase out that particular school. So that was a lost cause in a sense. But we have, 
you know, schools that had a D that we moved them with the targeted plan and the 
supports, they moved to an A - a D to a B, and again, I think it’s a numbers game - 
if you could just focus on targeting those specific kids who are very low and ensuring 
that they learn more, pick up more skills. 

In this Network Leader’s description, the point of customizing support was to help each school play 
a “numbers game.” From this perspective, the role of the Network was to help schools focus on 
“those specific kids who are very low” and figure out the right kinds of supports that they need to 
“learn more, pick up more skills.” This approach towards improvement conceived of school 
improvement as being focused on student-level data and was referred to as a “numbers game.” 

Several Network leaders suggested that a “numbers game” approach was an optimal strategy 
because it focused improvement efforts on matters that were more easily monitored and within 
principals’ locus of control. The Network Leader from CFN 703 explained that his goal was to help 
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principals identify the metrics on the Progress Report that principals could easily attend to. He 
explained his perspective on the numbers game: 

If you look at a progress report you’ll see there’s only a bar for social studies, well 
there’s two exams in social studies, there’s three exams in science, there’s three or 
four in math. So what they do is, and this is where it’s kind of unreliable in my mind, 
but what they do is they look at historical data trends for a given demographic and 
then they assign a weight to that particular student for each of the given exams and 
each exam the historical data is looked at separately. So a kid that comes in as a level 
one, that’s black, that’s free lunch, could have a weight of like nine. And then a kid 
that comes in at a level four that’s Asian and not free lunch might only get a decile of 
one or two. And then based on the data they make predictions about how if the kids 
are going to pass and what they’re going to pass with and then what the kid actually 
performs you’re measured against that, the average generates that prediction. So it’s 
not really progress in my mind. Again, this is just something that schools do and this 
is certainly something I did, is that, a lot of schools don’t do it, but smart schools do 
do it. You can find out precisely what every kid’s decile weight is in your school and 
so one of the things I did is, I would find that out. 

In this excerpt, the Network Leader from CFN 703 highlights how to play the numbers game and 
the tradeoffs involved in such a game. He described how the Progress Report incentivized schools 
to pay attention to student-level data - specifically, students’ decile weights and testing information. 
These numbers were significant determinants in the methodology of scoring on the Progress 
Reports. Also, present in this excerpt is the Network Leader’s doubts about the reliability of the 
methodology and the conflict it poses because in his mind, “it’s not really progress.” Yet, as he 
noted, “this is just something that schools do and this is certainly something I did.. .a lot of schools 
don’t do it, but smart schools do do it.” 

There were other Networks that expressed a similar belief - that it was “smart” for schools 
to focus on improving their grades on the Progress Reports and this was something that Networks 
could help schools with because the Progress Report was perceived by many to be technical. A 
cluster team member explained the general sense of the Progress Report by noting that, “.. .A lot of 
folks are incredible instructional leaders, which is an area where I sort of lack and need to grow, but 
those same folks are a bit apprehensive when they see the Progress Report because it’s not - it’s 
more of an organizational piece. It’s more about understanding the matrix and everything else that 
goes into it - so folks are turned off.” As this cluster team leader expressed, there was a view that 
the Progress Report required a proficiency with data and a particular kind of leadership skill set that 
was different from those who positioned themselves as instructional leaders in their schools. He had 
been a former economics and accounting major when he attended college - “so, the Progress 
Report and metrics and everything else that goes into it is sort of like second nature to me.. .1 was a 
math teacher; I am not fearful of the mathematics.” But, he noted that some principals “today have 
gone through the New York City Leadership Academy” where the focus is on developing leaders 
well equipped to help teachers. According to the cluster team member, this meant the leaders were 
not well positioned to deal with the math on the Progress Report. This diametric opposition 
between data management and instmctional leadership was seen as an opportunity for Networks to 
assist principals with understanding and responding to quantitatively-based Progress Reports in ways 
that would be advantageous for their schools. 

The Network Leader for CFN 701 explained that her team would help schools, “start to 
unpack it [Progress Report]. We’ll look at it and see if it was student progress - it usually is because 
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it’s sixty-percent is progress of students. So, that’s the big weight of the report.” One of the metrics 
of progress was how well a school contributed to its students’ progress - in particular, schools were 
evaluated on how well they served those students who performed in the lowest-third percentile in 
Math and ELA. Schools could receive extra credit when students in the lowest-third made 
significant gains (i.e., graduation or above 75 on state Regents exams). 

The Network Leader from CFN 701 explained that, . .usually, it’s just a matter of us trying 
to work with schools to target and support those troubled kids - or identify.. .we’ll help them take 
out.. .the lowest-third and then to ensure that they [principals] develop some system to sort of track 
and monitor the kids’ progress throughout the year - and get them targeted. Work on those targeted 
skills, cause that’s all it is. I think if you’re looking for progress, you’re going to focus on this and 
you’ll get your scores up.” The key for this kind of practice was that school scores would go up, if 
Networks could help principals determine which students they should monitor and target 
interventions to help. 

Depending on the circumstances of a school’s Progress Report, it was possible that this kind 
of strategy could improve schools’ scores on the Progress Report by a wide margin. One former 
Network Leader explained that when she worked as a principal, she was able to raise her school’s 
grade from a B to an A by requiring three students to go to summer school to earn additional 
credits. She described her experience, “In the middle of one night, I got up at three o’clock, I got a 
B, my old school. And I went in to look at the number how it says, how many out of 95 graduates, I 
noticed that those three students [who had gone to summer school] were not added in. So I emailed 
Jim [Chief Accountability Officer], three o’clock in the morning, said I can’t believe after a million 
conversations, these three students were not included in this file. And he emailed back, you are right. 
So my grade changed from B to A.” This story from her days as a principal earned this Network 
Leader credibility among the schools in her Network. She used this example as evidence to convince 
principals in her Network that a key improvement practice to be pursued was to focus on the 
indicators in the Progress Report that they could directly manage. 

This version of close monitoring and “educational triage” is slightly different than what 
Jennifer Booher-Jennings (2005) found happening for “bubble kids” in Texas schools. Booher- 
Jennings found that teachers and school leadership sought to create an appearance of school 
improvement, as would be evidenced with higher test scores, by diverting resources to “bubble kids” 
- those students on the threshold of passing the Texas Assessment of Knowledge and Skills test. 
What made this version different was that the networks were encouraging principals to attend to the 
lowest achieving kids, who in Booher-Jennings’ study were ignored. One former Network leader 
explained that this was the “beauty of the Progress Report.” He said. 

For years prior to the Progress Report you had your neediest, most at-risk students 
who were the ones that were receiving the least amount of opportunities in their 
school. So, for example, some of your worst teachers were the ones that would work 
with your neediest kids. If there were any additional dollars left over, hardly ever 
would those dollars go to supporting your neediest kids. And the Progress Report is 
predicated, if you want to get the maximum score possible, on helping your neediest 
kids. So, for me, that is a Godsend because the kids who need is the most are being 
given every opportunity possible to be successful. 

According to this former Network leader, the Progress Report was helping schools transform in 
ways that allowed them to serve better their neediest students. In his view, and others, focusing on 
the ‘numbers’ and student-level data was a strategy for focusing on equity. He said, “it should never 
be about gaming the system.. .but once you get them [principals] to embrace it [the Progress 
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Report], and get them to realize that you can make strategical decisions that really support your kids 
that really need support, they do embrace it and they’re able to run with it.” 

The networks that emphasized focusing on the Progress Report as an improvement strategy 
emphasized monitoring and managing improvement efforts at the student-level — tracking specific 
students and targeting interventions like extended day, tutoring programs, teaching assignments and 
monitoring student-level data to ensure that the lowest performing students were making progress. 
There were tradeoffs involved with this strategy. In reflecting on his Network’s approach to ‘moving 
schools,’ by focusing on the Progress Report, the Network Leader from CFN 703 acknowledged, “I 
don’t know how much you want everybody to do this because sometimes you lose insight on 
instruction.” When the Networks directed schools to focus on the Progress Reports, there could be 
quick results measured by indicators on the Progress Reports, but the efforts were more likely to 
consider improvement at the student-level - targeting specific kids’ experiences and less likely to 
focus efforts on school-level strategies that tried to transform school cultures, norms, routines, and 
structures to focus on improving instruction - this was the kind of strategy that emerged when 
Networks focused on the Quality Review as a guide for improving school performance. 

Using the Quality Review to develop new mindsets and change organizational 
conditions. Unlike the Progress Reports, which were annual evaluations of school performance 
based on student and quantitative data. Quality Reviews involved a two- or three-day school visit 
during which external reviewers would use a rubric to evaluate schools in three domains: 
instructional core across classrooms, school culture, and systems for improvement. The rules for 
when a school would receive a Quality Review varied along several factors, including a school’s prior 
performance on the Progress Report and state accountability metrics. Schools designated as Priority 
or Focus, based on New York State standards, were required to undergo a Quality Review more 
frequently. Generally, the rule was that every school would receive a Quality Review, at least, once 
every four years, but if a school earned the lowest score on a Quality Review, “underdeveloped,” 
then it would receive another review after having one year to improve. Also, it was common for 
schools that earned three consecutive C’s on the Progress Report, or lower grades for two 
consecutive years to be required to undergo a Quality Review. New schools in their first year were 
exempt from Quality Reviews that were attached to consequences, but they were still required to 
undergo a New School Quality Review (NSQR), which followed the same process and used the 
same rubric as full Quality Reviews. When Networks talked about using the Quality Review as a 
framework for defining improvement practices in schools, it was the mbric that served as the guide. 

The Networks that focused on the Quality Review rubric held two assumptions. The first 
was that school improvement practices should be focused on and directly related to instruction, and 
the second was that the practices outlined on the rubric were believed to be research-based practices 
that would strengthen teaching and learning across classrooms. Without being prescriptive, the 
mbric defined what “coherent curricula” across grades should look like. For example, a proficient 
school has coherent curricula that would be evidenced by, “school leaders and faculty ensure that 
curricula are aligned to the CCLS [Common Core Learning Standards], and/or content standards, 
integrate instructional shifts, and make purposeful decisions to build coherence and promote college 
and career readiness for all students” (Indicator 1.1a, NYCDOE, 2015). 

The rubric also offered an explanation of professional community among teachers by 
seeking to measure the extent to which teachers engaged in “structured professional collaborations 
on teams using an inquiry approach that promotes shared leadership and focuses on improving 
student learning” (Indicator 4.2, NYC DOE, 2014). These elements - coherent curricula across 
grades and strong professional communities are the kinds of mechanisms that policy scholars refer 
to as an instructional infrastmcture that provides guidance and promotes school-wide continuous 
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improvement practices (see, Byrk et al., 2010; Coburn, Hill, & Spillane, 2016). In most lines of 
implementation research, incoherence, inconsistent guidance, or uneven attempts to change 
classroom teaching are often explained by missing infrastructure, which speaks to the novelty of the 
Quality Review rubric, which, in some ways, provided an instructional infrastructure. 

Some Networks used the language of the mbric and reminded schools that they would be 
evaluated on how well they attended to the systems and practices related to the improvement of 
instruction in order to persuade principals to adopt what many referred to as a “quality review 
mindset” towards school improvement. 

A “quality review mindset,” as explained by several Network leaders was the view that 
improvement practice should be directed at whole-school organizational conditions related to the 
improvement of instruction. This may include having teacher teams meet regularly to discuss 
student work, or designing protocols to ensure that curriculum was aligned to standards across 
classrooms. The ‘mindset’ was also that improvement practices should be on-going commitments to 
the stmctures, routines and practices that were meant to improve the quality of instruction across 
classrooms. This was a departure from prior responses to accountability mandates that resembled 
what some Network Team members referred to as “dog and pony shows.” 

When principals put on shows for supervisors (i.e.,Superintendents), it often involved 
compiling evidence of improvement efforts in binders - things like agendas or copies of powerpoint 
presentations used during school meetings. In many ways, these artifacts were used to illustrate that 
the school was satisfying a mandate from the Supervisor, but the Quality Review required 
improvement efforts to be evident in active processes and lived practices - not just artifacts 
collected and stored in binders. The Quality Review demanded that schools demonstrate evidence of 
improvement of instruction during multiple, routine classroom observations and through interviews 
with teachers and other staff, students and parents. The NYC Department of Education had 
implemented the time-consuming and expensive Quality Review because it was trying to move away 
from incentivizing empty compliance-based activities like compiling binders of agendas that did not 
touch instmction or regular, day-to-day practices in schools. 

The point of the Quality Review was to encourage a culture in schools that focused on 
relations between the organizational stmctures and how the school collectively sought to improve 
instruction. One former architect of New York City’s accountability system, described the Quality 
Review as a “crucial tool for creating more incentive on the part of the principals to be strategic 
leaders.” According to this official, the DOE “didn’t tell them [principals] how to do it, but it said, 
come up with a plan. It’s your plan; you figure it out.” Because of the openness of the instmment, 
some Networks saw this as an opportunity to support principals with developing plans and engaging 
in on-going work, which allowed them to take an accountability tool, which in the past was a 
snapshot of school performance focused on student outcomes to embrace the Quality Review as an 
improvement strategy such that several members from different Networks referred to this approach 
as a “mindset.” 

This point is well illustrated by a Network team member from CFN 702 who explained that 
she felt the, “QR [Quality Review] process and QR rubric is one of the best instruments I prefer for 
professional development.. .1 see my job as a principal coach is to help principals who don’t have 
the time to fully understand what this rubric means, help them digest it in meaningful ways and help 
them start making those changes.” According to this network member, and others, “those changes,” 
included managing the school to change instmction. She continued, “And from my understanding, 
it’s a year-round process - it doesn’t start and it doesn’t stop - it’s just you always have to think 
about having a QR mindset.” A “QR mindset” is a particular approach to school improvement that 
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focuses on targeting routines and teachers’ opportunities to be about improving instruction across 
classrooms in ways reminiscent of what industry refers to as continuous improvement. 

The view that the Quality Review promoted a continuous improvement strategy was echoed 
by another DOE official, who while explaining the Quality Review described it by saying that its 
purpose was to promote, “a particular view of how schools should be organized, and that view is 
sort of like continuous improvement — sort of process view.” This view requires principals to do 
more than fill binders with artifacts as proxies of improvement efforts. Rather, this view required a 
different conception of improvement practice as linking organizational conditions to classroom 
instruction in ways that would be evident when external reviewers walked through classrooms. 

To accept this view, some principals had to not only learn about the Quality Review, but 
they also had to unlearn their prior conceptions of improvement practice as a compliance-based 
task. Network team members from CFN 702 and 704 described working with principals towards 
these ends, which in some cases was welcomed by principals and in other instances, this created a 
tension that Networks had to manage in order to help schools develop a quality review “mindset.” 

Tensions emerged for those networks that tried to use the Quality Review as the key support 
- including what it meant to try to balance intra-school politics to balance budgets. Another tension 
involved the various meanings of components involved in the rubric as it applied to a school’s 
circumstances. For example, one of the indicators required schools to “design engaging, rigorous, 
and coherent curricula” (Indicator 1.1, NYCDOE, 2015). Debates could ensue as to what 
constitutes “rigorous habits” and “higher order skills” for fourth graders as compared to eighth. 

This would create conflict for a network; networks who pursued this long term, highly subject 
process risked jeopardizing their access to a school. 

For example, the Network leader from CFN 701 explained that conflict would arise when 
schools ranked themselves as “well developed” on indicators of the Quality Review during their self- 
assessments, but the Network team evaluated school as “developing,” which is a lower score. The 
Network Leader from CFN 701 said, “And many times, they [principals] sit in a room, we score. We 
come back and say, okay, 1.1, what do you think you are? And they’ll say, ‘well-developed.’ And the 
network will say, ‘developing,’ and there you go, conflict.” She went on to explain that she believed 
one school left her network because this kind of exchange occurred four times: 

And at one school, I’m sure that’s why she [principal] left.. .we went back, I think it 
was four times, and they were insisting that they were well-developed. So, the second 
time, the place was filled with stuff - folders, documents, ridiculous. I had to tell 
them, ‘Stop! Don’t bring anymore in here, because the documents were digging them 
worse into the hole because it’s a show. Sorry this is not a good artifact.’ So, it can be 
a tension - especially, if people aren’t honest about reflective about the work that is 
in front of them. 

In this excerpt, the network leader makes clear that there was tension for her Network when it tried 
to use the Quality Review as an improvement strategy because it required a new kind of relationship 
between a support provider and school. What was new was that the school and provider needed to 
be able to come to an agreement about the “work in front of the school” as would be measured by 
the Quality Review process, which required external reviewers to find evidence of certain 
commitments - commitments to implementing the Common Core Standards, the school’s view of 
pedagogy, inclusive, but rigorous instruction in multiple classrooms, as well as, evidence located in 
school routines related to the improvement of instruction. For Networks, focusing on this kind of 
improvement came with risk because principals may respond to tensions by leaving a network - as 
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was the case for the principal that disagreed with the leader from CFN 701 about evidence of 
improvement efforts living in binders instead of classroom instruction. 

A Network team member from CFN 702 also referred to the tension that existed for 
networks trying to use the Quality Review as a learning tool. She explained that her network learned 
that, 

.. .our network was one of the few giving critical feedback and most of the other 
networks were giving only accolades and ‘well-develops’ and giving their principals 
pats on the back, whereas, we always felt like it [Quality Review] was an opportunity 
to learn and approached it that way, but it meant that our principals got upset 
because they heard their other friends being praised, and not that they weren’t 
praised, but we just took a different approach. And I feel like the tension is that 
networks may be feeling a tension around giving critical feedback to the schools that 
are rating them. And I think therein lies a rub that we haven’t, as a network, cared 
about - like we care about making the school better and I think I’ve heard from 
other teams this tension of making sure they’re serving principals in ways that their 
principals are happy. 

In this excerpt, the Network team-member notes the challenges Networks faced in trying to use the 
Quality Review as a learning tool, but their goal, despite these challenges was to use the mbric as a 
way to develop mutual understandings between the school and Network about what priorities for 
improvement should be; such a strategy positioned the Network as a critical friend trying to help the 
school understand the rubric and shift ideas about how to use an accountability instmment as an 
improvement strategy. 

One senior official from the New York City Department of Education explained that 
treating accountability like an improvement strategy was the initial intent for DOE as the networks 
were meant to be service providers that were also held accountable for schools’ outcomes. The type 
of service they imagined was using the Quality Review to drive conversations and cultivate 
continuous improvement commitments to improvement. He explained, “If the network team takes 
that day review very seriously, and has, and this is tricky business, where you have standing with the 
principal and the school community, where you can go in for a day and give some hard feedback - 
not just glowing feedback and manage to build on that, then you have a real tool that’s an 
improvement tool.” The “tricky business” for Networks that wanted to use the Quality Review as a 
strategy because they risked upsetting principals, who were free to leave the Network to find one 
where efforts to improve a school’s outcomes would involve less invasive tactics than attending to 
school-level routines and practices. 

There is convincing evidence of principals, generally, attempting to buffer against external 
intrusion in matters related to instmction (Elmore, 2000; Hatch & Honig, 2004). When a principal 
was used to buffering, as may have been permitted or incentivized under previous school support 
arrangements, it presented issues for the Networks like CFN 701 and CFN 702 that wanted to 
promote an innovative approach to school improvement that relied mutual partnerships between 
the Networks, outsiders, and the inner-workings of schools. All Networks were dependent on 
principals providing access to the school. Networks like CFN 701 and 702 wanted a particular type 
of access to the instructional core and organizational conditions that were outlined on the Quality 
Review rubric like routines for improving instmction. This was a risky and novel endeavor for many 
schools that were potentially more comfortable being invisible under the old support arrangements. 

To broker mutual relationships with schools, CFN 702 and CFN 701 relied on the Quality 
Review rubric as an instrument used in negotiations with schools to establish shared commitments 



Education Policy Analysis Archives Vol. 25 No. 122 


22 


to particular strategies for school improvement. For example, CFN 702 developed a routine and 
additional tools based on the Quality Review rubric. One tool they created was a protocol called, 
“Your School, Your Story” that focused on three indicators from the rubric: Goals and action plans 
(Indicator 3.1), Curriculum (Indicator 1.1) and Pedagogy (Indicator 1.2). A member from the 
network team explained how CFN 602 used this the “Your School, Your Story” protocol during 
monthly principal conferences that joint meetings with the 18 principals in the network. She shared. 
We meet at a different school each month. I would fishbowl with a conversation 
with the principal around 3.1, 1.1, 1.2 [Quality Review rubric indicators] and we’d 
spend forty-five minutes where the principals got familiar with the kinds of questions 
the reviewer asks in those three areas and really give the principal a chance to tell. 

We kept referring to, the story of their school. Who they are, what they’re about, 
what this looks like in terms of a student’s experience in the classroom in their 
school. Other principals would take notes on what the principal’s saying, and, at the 
end of the conversation, before we went into classrooms, we’d clarify what we heard 
the principal say in terms of what are the tilings happening in the classrooms that are 
distinct to your school. 

In this excerpt, the team member illustrates the social, routine practice that the Network created that 
was based specifically on the indicators of the rubric, but allowed each of the schools to create their 
individual access points and self-directed focus for improvement efforts. After these meetings, the 
team of principals would walk through the classrooms and observe instmction in the classrooms the 
host site. After the walkthrough, the principals would debrief during which the host principal would 
have a chance to revise her story and adapt plans based on these discussions. The plans were then 
available to be used potentially by the principal and the network team that offered specific supports 
like coaching for teacher teams or curriculum mapping. 

Summary and Policy Implications 

Evidence from this study reveals relations between novel accountability metrics and 
commitments to school improvement practices. Unlike other data-use studies that are limited to 
how educators make sense of test-based data to drive instmctional improvement, this study presents 
evidence how improvement practices within schools was linked to negotiations and how shared 
commitments to particular improvement strategies were formed between adaptive support providers 
that were seeking to customize support and schools and how accountability metrics shaped these 
negotiations. 

Among the four network teams that I examined, I found evidence that these teams used 
NYC’s accountability tools, the Progress Reports and Quality Reviews to orient their efforts to 
develop customized, targeted plans, and that depending on which tool the Network privileged, 
certain improvement strategies and priorities were favored. 

For those networks that championed the Progress Reports, improvement activities were 
focused on monitoring student-level data with the intent of targeting specific students for 
interventions like summer school or after-school tutoring. This was a short-term approach that 
because of the methodology used to calculate scores on the Progress Reports would enable some 
schools to make substantial gains on Progress Report within a year. By contrast, those networks that 
championed the Quality Review mbric as a framework for school improvement, designed support 
structures like routine meetings with principals that targeted whole-school continuous improvement 
of instruction. This approach took the long view and was predicated on the theory that focusing 
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collective efforts on the instructional core - interactions between students, teachers and content 
would eventually lead to improved students’ test scores and higher marks on school accountability 
metrics. These vastly different approaches towards school improvement were both incentivized by 
the accountability scheme that held CFNs accountable for their schools’ performances. 

It is not surprising that focusing on different tools produced different kinds of improvement 
practices. In their study of how districts designed architectures for learning for teachers implanting a 
new math curriculum, Stein and Coburn (2008) different forms of learning emerge from the use of 
different tools. What the tools specify and who interacts with the tool in particular kinds of 
conditions influenced the designs for practices and people’s responses to particular designs. How 
meaning is negotiated is directly dependent on the balance of reification, or what is specified in a 
tool, and participation, or social interactions from which meaning learning may emerge (Stein & 
Coburn, 2008; Wenger 1998). Drawing on Wenger, Stein and Coburn (2008) make the point that, 
“different reifications provide different affordances for the negotiation of meaning” (p. 615). 

This point of variation is an important one that has plagued policy makers seeking to 
encourage particular types of school reform by creating accountability policies that issue particular 
tools to measure quality - predominately, test-based models that hold schools accountable for 
students’ absolute performance on standardized tests and treat all schools, as if they possess skills, 
orientations, and resources to respond with adequate diagnoses and prognoses. Accountability 
policies like NCLB relied on standardization to promote increased quality amongst various local 
contexts. The theory of action was that specifying the standard, like all students must be proficient 
in math and reading else a consequence would be enforced, would remedy the problems plaguing 
underperforming schools. But standardization of quality metrics means that variation occurs in 
pursuit of quality, and depending on the metrics may require schools to learn how to perform 
drastically different practices, or to achieve the metric, it may require very little change due 
preexisting circumstances that may not actually be attributed to the behaviors or performances of 
adults working in any given school. While the policy landscape is shifting to embrace progress 
metrics that actually try to measure what a school contributes to student learning, there is still the 
persistent problem of uneven improvements within and between schools, which suggests we need 
better improvement strategies and, perhaps, better metrics (see, Finnigan, O’Day & Bitter, 2009). 

Within schools, test-based metrics can lead educators to focus resources and efforts on 
those students designated as bubble kids- those most likely to make gains on tests (Jennings, 2012), 
or narrow focus on tested subjects (Diamond & Spillane, 2007). At the district level, research has 
largely shown that when strict accountability policies are introduced by a state or district, those 
schools that already possessed strong internal practices for improving instruction were best 
positioned to respond to the pressures meant to incentivize internal routines — especially for schools 
that lagged behind; earlier studies suggest that policies typically fail to penetrate practice where 
change is most needed (Elmore et al., 2004; Newmann et al., 1997). 

The evidence presented here has a number of implications for district leaders and policy 
makers. With the passage of the Every Student Succeeds Act in 2015, State and district leaders were 
granted an opportunity to design new accountability systems that include new metrics and support 
structures. This study signals that perhaps accountability metrics can penetrate schools’ internal 
commitments to school-wide improvement — regardless of the school’s prior performance or the 
resources available within in a school when support providers are made available that are also held 
accountable for outcomes. 

Because the accountability scheme in NYC under Mayor Bloomberg was applied to support 
providers in perhaps a close approximation to what Richard Elmore referred to as reciprocal 
accountability, the metrics figured prominently into how these teams developed joint plans with 
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schools. There was no bureaucratic arrangement that required engagement; rather, successful 
engagement depended upon negotiations and the extent to which these were mutually beneficial for 
Networks and schools. What was negotiated were the processes both sides interpreted as being 
necessary to achieve quality; together, the school and Network had to attend to what would be 
needed to measure up to what was specified by the metrics. This point was echoed by one Cluster 
team member who credited the accountability scheme as changing the nature of school support: 

.. .1 think, again, that because it’s part of the network accountability, they pay closer 
attention and I do think that they organize their coaching support to make sure that 
systems and structures in schools are strengthened.. .they’re even more invested in 
getting the process right, making sure that it’s meaningful investment of time for 
both them and the schools. 

As the excerpt demonstrates, the accountability scheme created the motivation for support 
providers to “pay closer attention” to the tools and make sure that their efforts to work with schools 
would lead to improved outcomes on the Progress Reports and Quality Reviews. Finding ways to 
diagnose and design solutions through collaborative engagement was a “meaningful investment of 
time for both them and the schools,” which is a closer approximation of mutual support partnership 
than typical support stmctures where providers are seen as teachers providing professional 
development and resources and schools are supposed to absorb information and change practices in 
order to improve outcomes. 

It is possible that new forms of improvement practices could emerge under novel 
accountability systems. This research suggests that whether new forms of improvement practices 
emerge may depend on the designs for accountability tools and the contexts within which providers 
and schools are tasked with collaborating to diagnose and craft improvement strategies. Lastly, this 
exploration of the CFNs in New York City also suggests that part of the context that may matter is 
the extent to which providers and schools are both held accountable for results. 
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